Multi-Modal Tracking of Faces for Video Communications

visual processes in cyclic manner. This paper describes a system which uses multiple visual processes to detect and track faces for video compression and transmission. The system is based on an architecture in which a supervisor selects and activates visual processes in cyclic manner. Control of visual processes is made possible by a confidence factor which accompanies each observation. Fusion of results into a unified estimation for tracking is made possible by estimating a covariance matrix with each observation. This system demonstrates that robust operation can be achieved by coordinating multiple visual processes. Control of individual processes is made possible by the inclusion of a confidence factor accompanying each observation. Fusion of the results is made possible by the determination of an error estimate (a covariance matrix) for each observation. Composing a system from a redundant ensemble of processes permits the overall system to automatically adapt to a variety of operational circumstances. Visual processes for face tracking are described using blink detection, normalised color histogram matching, and cross correlation (SSD and NCC). Ensembles of visual processes are organised into processing states so as to provide robust tracking. Transition between states is determined by events detected by processes. The result of face detection is fed into recursive estimator (Kalman filter). The output from the estimator drives a PD controller for a pan/tilt/zoom camera. The resulting system provides robust and precise tracking which operates continuously at approximately 20 images per second on a 150 megahertz computer work-station. The following section reviews the visual process architecture used for the system and describes techniques for estimating a confidence factor and an error bounds for visual processes. A tracking process based on a zero-th order recursive estimator is described. Visual processes are described for the detection and tracking of faces using blink detection, color, and correlation, as well as processes for estimation and camera control. Visual processes are grouped into states with state transitions triggered by events. An example of an execution trace is provided with the system as configured at the time of writing.